Skip to content

fix: optimize_projections failure after mark joins created by EXISTS OR EXISTS#21265

Merged
berkaysynnada merged 4 commits intoapache:mainfrom
buraksenn:fix-optimize-projections-fail
Apr 17, 2026
Merged

fix: optimize_projections failure after mark joins created by EXISTS OR EXISTS#21265
berkaysynnada merged 4 commits intoapache:mainfrom
buraksenn:fix-optimize-projections-fail

Conversation

@buraksenn
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Issue has details but main problem is mark columns from LeftMark joins leak into parent join schemas, causing optimize_projections optimizer to fail.

What changes are included in this PR?

Add a projection after embedded subquery decorrelation to strip mark columns, following the same pattern as scalar_subquery_to_join. I've seen this projection is merged in the final plan.

Are these changes tested?

Added test case for reported failure

Are there any user-facing changes?

No.

@github-actions github-actions bot added the optimizer Optimizer rules label Mar 30, 2026
Copy link
Copy Markdown
Contributor

@neilconway neilconway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall lgtm!

Comment thread datafusion/optimizer/src/optimize_projections/mod.rs
Comment thread datafusion/optimizer/src/optimize_projections/mod.rs Outdated
Comment thread datafusion/optimizer/src/optimize_projections/mod.rs Outdated
Comment thread datafusion/optimizer/src/optimize_projections/mod.rs Outdated
@neilconway
Copy link
Copy Markdown
Contributor

@alamb This PR looks reasonable to me.

Copy link
Copy Markdown
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me too - thanks @buraksenn and @neilconway

Copy link
Copy Markdown
Contributor

@berkaysynnada berkaysynnada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've finally had a chance to take a look at this. LGTM, it's a clear fix

@berkaysynnada berkaysynnada added this pull request to the merge queue Apr 17, 2026
Merged via the queue into apache:main with commit d1800db Apr 17, 2026
35 checks passed
@berkaysynnada berkaysynnada deleted the fix-optimize-projections-fail branch April 17, 2026 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

optimize_projections fails after mark-join involved

4 participants